Cubical Gamut Mapping Colour Constancy

A new color constancy algorithm called Cubical Gamut Mapping (CGM) is introduced. CGM is computationally very simple, yet performs better than many currently known algorithms in terms of median illumination estimation error. Moreover, it can be tuned to minimize the maximum error. Being able to reduce the maximum error, possibly at the expense of increased median error, is an advantage over many published color constancy algorithms, which may perform quite well in terms of median illumination-estimation error, but have very poor worst-case performance. CGM is based on principles similar to existing gamut mapping algorithms; however, it represents the gamut of image chromaticities as a simple cube characterized by the image’s maximum and minimum rgb chromaticities rather than their more complicated convex hull. It also uses the maximal RGBs as an additional source of information about the illuminant. The estimate of the scene illuminant is obtained by linearly mapping the chromaticity of the maximum RGB, minimum rgb and maximum rgb values. The algorithm is trained off-line on a set of synthetically generated images. Linear programming techniques for optimizing the mapping both in terms of the sum of errors and in terms of the maximum error are used. CGM uses a very simple image pre-processing stage that does not require image segmentation. For each pixel in the image, the pixels in the Nby-N surrounding block are averaged. The pixels for which at least one of the neighbouring pixels in the N-by-N surrounding block differs from the average by more than a given threshold are removed. This pre-processing not only improves CGM, but also improves the performance of other published algorithms such as max RGB and Grey World. Introduction An image of a scene depends on the camera characteristics, the reflectances of the surfaces in the scene and on the scene illuminant. For many computer vision tasks such as object recognition or image tracking, it is desirable to remove the effects of variations in the chromaticity of the illuminant. Illumination-estimation algorithms offer one of many ways of approaching the problem. They attempt to estimate the chromaticity of the scene illuminant, often assuming the presence of only a single source of illumination, since once the chromaticity of the illuminant is known, the chromaticity of reflectances of the surfaces in the scene can be calculated. There are many simple algorithms, such as MaxRGB [1], Grey World [2], Shades of Grey [3] and Grey Edge [4] which rely on various assumptions about the scene. MaxRGB assumes that the chromaticity of the maximum R, G, and B values in the image correspond to the chromaticity of the illuminant. Grey World assumes the average of RGB values in the scene is achromatic. Shades of Grey assumes the p Minkowski norm of the RGB values to be achromatic. The Grey Edge algorithm generalizes the assumption even further by assuming that p Minkowski norm of the derivatives of RGB values in a scene is achromatic. Despite their simplicity, some of these algorithms—Shades of Grey and Grey Edge in particular— have been shown to perform quite well for real-world images. More sophisticated algorithms use statistical knowledge about the frequency with which different reflectances and illuminants occur in the real world. Neural Networks [5], Color by Correlation [6] and various gamut mapping algorithms, such as Forsyth’s original algorithm [7] and Gamut Constrained Illuminant Estimation (GCIE) [8] fall into this category. These algorithms build statistical models of the distribution of RGB and/or rgb chromaticity values (r = R/(R+G+B), g = G/(R+G+B), b = G/(R+G+B)) in the image and then evaluate the input image relative to the statistical model of the world. Of these algorithms, GCIE [8] is, to our knowledge, currently one of the best performing color constancy algorithms. Other color constancy algorithms exploit physical principles of image formation and often use Shafer’s dichromatic model of reflectance [9]. Algorithms such as [10] and others rely on the presence of specularities or interreflections. The performance of the algorithms improves when combined with additional statistical knowledge [12] or constrained to a set of known illuminants [11]. This paper introduces a new color constancy algorithm called Cubical Gamut Mapping (CGM). CGM is computationally very simple, yet performs better than many currently known algorithms in terms of its median illuminationestimation error. It can also be tuned to minimize the maximum error. CGM combines statistical knowledge about the distribution of reflectances and illuminants in the real world with a simple assumption about the scene. In terms of its statistical model, CGM is based on principles similar to gamut mapping algorithms except that it represents the gamut of image chromaticities as a simple cube characterized by the image’s maximum and minimum rgb chromaticities, rather than their more complicated convex hull. It also uses maximum RGB as an additional source of information about the illuminant. The estimate of the scene illuminant is obtained by linearly mapping the chromaticity of the maximum RGB, minimum rgb and maximum rgb values. The algorithm is trained off-line on a set of synthetically generated images. Linear programming techniques for optimizing the mapping in terms of both the sum of errors and in terms of the maximum error are used. Color constancy algorithms for real world images rely on image pre-processing. Pre-processing is usually done in a separate step before the color constancy algorithm is invoked to reduce noise and to reduce the colour shifts created by chromatic aberration or demosaicing. It has been shown [13], that the pre-processing can significantly improve the performance of color constancy algorithms. Finlayson [19] suggested that further performance gains can be obtained by introducing new pre-processing algorithms. CGM uses a very simple image pre-processing stage that does not require image segmentation. For each pixel in the image, the pixels in the N-by-N surrounding block are averaged, and then pixels for which at least one of the neighbouring pixels in the N-by-N surrounding block differs 466 ©2010 Society for Imaging Science and Technology from the average by more than a given threshold are removed. Dark pixels are also removed. This pre-processing not only improves CGM, but also improves the performance of other published algorithms such as max RGB and Grey World. The paper is structured as follows. The pre-processing algorithm used by CGM is described in section 2. Section 3 introduces CGM. Section 4 discusses the results obtained by running CGM on the SFU test set of images [14]. Section 5 concludes. Image Pre-processing The image pre-processing algorithms try to address problems introduced by random pixel noise such as shot noise and colour shifts created by chromatic aberration and demosaicing artifacts. Generally, color constancy algorithms assume there is only a single scene illuminant. Hence secondary sources of illumination and interreflections present in the scene may affect the performance of the algorithms. We make the assumption that the darker parts of the image are more likely than the brighter parts to be illuminated by a secondary illuminant and so the pre-processing removes them from further consideration. For example, objects illuminated by the sky alone are darker than objects that are illuminated directly by the sun. Removing very dark pixels eliminates the cases where camera noise and quantization errors may be high; however, we found additional improvement by removing less dark pixels as well, and attribute the improvement to a reduction in the effects of secondary illuminants. As Barnard showed [13], the illumination-estimation errors measured in RMS distance in rg-chromaticity space (r = R/(R+G+B), g = G/(R+G+B)) may vary by as much as 30% depending on the pre-processing method used. As an example, consider the well known MaxRGB color constancy algorithm which computes the rgb of the illuminant by finding the maximum R value in the whole image, maximum G value and maximum B value and then normalizing the result. The MaxRGB RMS distance in rg-chromaticity space error ranges from 0.053 to 0.79 depending on the pre-processing method [13]. Existing Pre-processing Methods Some of the pre-processing methods reported in the literature are: Region-growing. Region-growing has been used by Barnard [13], as well as by many other authors, to stabilize the RGB values used in subsequent colour constancy processing. The region-growing was subjected to two constraints: chromaticities in a region were within an absolute threshold of each other, and the brightness values in a region were within a relative threshold of each other. Regions that contain less than a given number of pixels are discarded. Once the regions were found, the averages of the RGBs in each region are used as input to the color constancy algorithms. Block averaging. The RGBs of pixels within fixed-sized, non-overlapping blocks are averaged and the resulting average RGB values are used in subsequent processing [13]. A variation on the block-averaging method checks the variance and discards those image blocks having high RGB variance. Binarized RGB [13]. This technique distributes blocked averaged results into discrete bins in RGB space, with 100 discrete values per channel resulting in a total of 1,000,000 bins. The average RGB value from each bin is used as the preprocessing output. Note, that binarized RGB technique removes the information about the number of pixels in each of the bins and simply reports the presence or absence of each of the 1,000,000 possible colors. Gaussian smoothing. Gaussian smoothing reduces the noise and is used as a pre-processing step for the Grey Edge method [4]. Dark pixel removal. Dark pixels, those with R + G + B < Threshold are removed. Dark pixels tend to be adversely affected by camera and quantization noise. As well, brighter pixels have a higher chance of being directly illuminated by the main illuminant. Dark pixel removal was used, for example, in [15]. Proposed Pre-Processing—Even Blocks We propose a pre-processing method that is similar to dark-pixel removal combined with a modified block averaging strategy. Each pixel is considered relative to its N-by-N neighbourhood of surrounding pixels. If the R, G, B and r, g, b code values of each surrounding pixel do not differ from the corresponding means of the R, G, B and r, g, b code values in the neighbourhood by more than some threshold amount then the central pixel is kept and the output (R, G, B) for the central pixel is set to be the mean code value of the pixels in the neighbourhood. If any one of the R, G, B or r, g, b values from any of the surrounding pixels differs from the R, G, B or r, g, b means by more than the given threshold, then the central pixel is flagged as unreliable in the output. A color constancy algorithm that uses the pre-processed output then simply disregards all such flagged pixels. In our experiment, R, G, B code values are first normalized to the range [0, 1]. We used N = 5 and a threshold of 0.1. The threshold for the removal of dark pixels is set to the mean image intensity, where intensity is computed as (R+G+B)/3. The effect of the algorithm is to reduce noise by averaging pixels from within an N-by-N block. The algorithm removes pixels whose neighbourhoods vary in any of the channels R, G or B or any of the chromaticities r, g, or b. Therefore, the algorithm tends to discard pixels near sharp edges, and thus removes pixels that might be affected by chromatic aberration. The algorithm preserves more color variation in the scene than segmentation because the regions from which the average is computed are limited in size. Also note that segmentation may not remove chromatic aberrations along longer edges because the pixels along the edge may collectively form a large enough region to be retained. Gaussian smoothing and block averaging both tend to reduce the effects of chromatic aberration by averaging edge pixel values with their neighbours, but do not eliminate chromatic aberration’s effects completely. The proposed algorithm has a better chance of preserving smaller patches of color and smaller specular highlights than block averaging with the variance check. This is because the proposed algorithm does not divide the image into non-overlapping blocks whose boundaries may split smaller patches of color. Rather, it centers a block on each pixel. Results The pre-processing algorithm has been tested with MaxRGB and Grey World on the SFU color constancy test set [14]. The set consists of 321 16-bit images of 30 scenes containing both matte and specular objects illuminated by 11 different illuminants (the test-set authors removed a few of the 11 x 30 possible scene/light combinations). The performance of CGIV 2010 Final Program and Proceedings 467 the algorithms is measured in terms for median angular error computed in the rgb sensor space of the camera. The MaxRGB algorithm combined with the new pre-processing technique returns a median angular error of 3.0 degrees. To our knowledge, the best reported result for the MaxRGB algorithm on the same test set using the same error metric so far has been 4.0 degrees [15]. Similarly, the pre-processing algorithm reduces Grey World’s median angular error to 6.0 degrees from 6.8 degrees when simple Gaussian Smoothing is used [4]. Cube Gamut Mapping Algorithm We show a computationally very simple color constancy algorithm that, when combined with the pre-processing introduced in the previous section, performs very well in comparison to some of the best color constancy algorithms. Gamut mapping colour constancy [7] reformulated to include constraints based on considering only a fixed set of illuminants (GCIE) [8]) is one the best performing color constancy algorithms for which there are published results on the SFU test set of 321 images. For testing on this set, GCIE selects from 87 possible illuminants, 11 of which are ones used in the test set. Gamut mapping exploits the relationship between the gamut of chromaticities found in a scene and the gamut of possible chromaticities for all known reflectances under a given illuminant [7]. Gamut mapping algorithms are based on the fact that a chromaticity gamut can be modeled as a convex structure. It has been shown, that illuminating a convex set of reflectances results in a convex set of RGB responses, as well as a convex set of rgb chromaticities [7][8]. In the algorithm proposed here, the modeling of the gamut is simplified. Instead of computing the gamut as the convex hull of the observed sensor responses, the gamut is represented by the smallest cube in rgb space containing all chromaticities in the image. The bounding cube is found by simply determining the largest and smallest values of r, g and b (separately) in the image. In addition to the gamut of the observed colors, the algorithm also uses the maximum R, G, B values from the image. It is apparent from the excellent performance of the MaxRGB algorithm shown in the previous section, that the 3 maximal R, G and B values carry a lot of potentially useful information about the scene illuminant. The chromaticity of the scene illuminant is estimated by a linear mapping transforming a column vector P of 9 input values P i to a column vector O of 3 output values O j corresponding to the rgb of the illuminant. The 9 input P i values are the chromaticity of the maximum RGB values, the maximum rgb values, and the minimum rgb values. The mapping is represented by a 9-by-3 matrix W. The output rgb O is computed as